security

Can commands be written to a logfile before they were executed in any shells?

Given the other questions, I take it that another way to think about the question might be "What can I do to make sure that I get a record of every command that has been executed?"

The accounting tools provided by the auditd daemon, developed by RedHat but available for any Linux, can do this. The software provides mechanisms to record not only commands that have been executed, but also resources consumed, session lengths, use of SUID binaries, change of important system files, changes in file permissions, changes in the system time, etc, etc, etc. It is comprehensive, able to watch both files and system calls. It can even be turned on in the kernel at boot time with the argument, audit=1. It doesn't matter if the executed commands were from the console, from an SSH session, or from a daemon. It's log files are usually configured to be stored in /var/log/audit, and if you are looking for executed commands, they could be found by using grep exec /var/log/audit/audit.log, for instance.

Generic Concepts on Security and Logging

The rest of this is out of scope for the question, but maybe important because of the security tag on the question or future readers. You can get a good overall security baseline configuration that will probably be mostly relevant for your Linux system(s) via the DISA RHEL-6 STIG (unofficial link given for convenience of viewing), and there are dozens of audit-related entries that you might use for your systems as well. The precise implementation of the concepts may differ among the major distributions of Linux but remain useful as summaries of real baseline security practices. (DISA offers many other good baselines for many other systems and software products, too.)

Consider that, although the monitoring and logging is fantastic, will you really keep up with the task of reading every line of the logs and also effectively associate one event with another in your mind as the messages continue to flow? Consider, too, that the integrity of the logged data is a prime importance. Can you trust that the one system, the very system on which the accounting software has been installed and configured, will never be compromised? What about revelations like Shellshock?

Thus, we should try to mitigate the tampering of log file data and use software to help us analyze and respond to the information in the logs.

One strategy to keep the integrity of the logged data is to make sure that all logging is stored not only locally, but also remotely via forwarding to a central logging server.

Make sure that the Log Server will only accept traffic from the Production Server on the port used by the logging software, like TCP/UDP 514 for syslog. Make the management network as private as possible using any available means, like using a private network (RFC 1918) for the management network. Implement VLAN IDs if possible and deny-all-allow-by-exception firewall configurations, of course. Give the network as much bandwidth as possible and utilize it as little as possible, perhaps even going so far as to engineer a separate "syslog network" because, depending on the number of devices in the network, a central syslog server can become over-loaded with syslog messages. In turn, this can cause the messages to become discarded when using UDP. You might think to use TCP instead, but this too can become a problem with syslog messages stacking up in queue on the syslog clients until all disk space has been consumed, effectively "shooting yourself in the foot." (Thus, it is sensible to create a separate partition or logical volume to define the maximum disk space available for local log files and to prevent run-away disk usage on the system partitions or volumes.) If we are using rsyslog, then, in addition to specifying the central logging server in the rsyslog client, the audispd plugin must be enabled in rsyslog in order to forward the logging related to accounting.

Why would we log anything at all for security's sake if we do not intend to rigorously inspect and potentially respond to the information? That alone can become a daunting task. Thus, the choice of log file analysis software is also important. It should be able to know, or to be configured to know, what constitutes a threat and how to present it to us for our awareness, that we might respond in a timely way. Otherwise our security logs are not much use, except maybe in post-incident analysis. The software should be able to analyze the data on the central log server for all the devices that have been configured to send syslog data to it - servers, workstations, routers - or whatever device can send the logging data.